从过去的经验中发现有用的行为并将其转移到新任务的能力被认为是自然体现智力的核心组成部分。受神经科学的启发，发现在瓶颈状态下切换的行为一直被人们追求，以引起整个任务的最小描述长度的计划。先前的方法仅支持在线，政策，瓶颈状态发现，限制样本效率或离散的状态行动域，从而限制适用性。为了解决这个问题，我们介绍了基于模型的离线选项（MO2），这是一个脱机后视框架，支持在连续的状态行动空间上发现样品效率高效瓶颈选项。一旦脱机而在源域上学习了瓶颈选项，它们就会在线转移，以改善转移域的探索和价值估计。我们的实验表明，在复杂的长途连续控制任务上，具有稀疏，延迟的奖励，MO2的属性至关重要，并且导致性能超过最近的选项学习方法。其他消融进一步证明了对期权可预测性和信用分配的影响。
translated by 谷歌翻译
Inspired by progress in large-scale language modeling, we apply a similar approach towards building a single generalist agent beyond the realm of text outputs. The agent, which we refer to as Gato, works as a multi-modal, multi-task, multi-embodiment generalist policy. The same network with the same weights can play Atari, caption images, chat, stack blocks with a real robot arm and much more, deciding based on its context whether to output text, joint torques, button presses, or other tokens. In this report we describe the model and the data, and document the current capabilities of Gato.
translated by 谷歌翻译
2021-12-09
对于在现实世界中运营的机器人来说，期望学习可以有效地转移和适应许多任务和场景的可重复使用的行为。我们提出了一种使用分层混合潜变量模型来从数据中学习抽象运动技能的方法。与现有工作相比，我们的方法利用了离散和连续潜在变量的三级层次结构，以捕获一组高级行为，同时允许如何执行它们的差异。我们在操纵域中展示该方法可以有效地将离线数据脱落到不同的可执行行为，同时保留连续潜变量模型的灵活性。由此产生的技能可以在新的任务，看不见的对象和州内转移和微调到基于视觉的策略，与现有的技能和仿制的方法相比，产生更好的样本效率和渐近性能。我们进一步分析了技能最有益的方式以及何时：他们鼓励定向探索来涵盖与任务相关的国家空间的大区域，使其在挑战稀疏奖励环境中最有效。
translated by 谷歌翻译
密集对象跟踪，能够通过像素级精度本地化特定的对象点，是一个重要的计算机视觉任务，具有多种机器人的下游应用程序。现有方法在单个前向通行证中计算密集的键盘嵌入，这意味着模型培训以一次性跟踪所有内容，或者将它们的全部容量分配给稀疏预定义的点，交易一般性以获得准确性。在本文中，我们基于观察到给定时间的相关点数通常相对较少，例如，探索中间地面。掌握目标对象的点。我们的主要贡献是一种新颖的架构，灵感来自少量任务适应，这允许一个稀疏样式的网络在嵌入点嵌入的关键点嵌入时的条件。我们的中央发现是，这种方法提供了密集嵌入模型的一般性，同时提供准确性更加接近稀疏关键点方法。我们呈现了说明此容量与准确性权衡的结果，并使用真正的机器人挑选任务展示将转移到新对象实例（在课程中）的能力。
translated by 谷歌翻译
我们研究了复杂几何物体的机器人堆叠问题。我们提出了一个挑战和多样化的这些物体，这些物体被精心设计，以便要求超出简单的“拾取”解决方案之外的策略。我们的方法是加强学习（RL）方法与基于视觉的互动政策蒸馏和模拟到现实转移相结合。我们的学习政策可以有效地处理现实世界中的多个对象组合，并展示各种各样的堆叠技能。在一个大型的实验研究中，我们调查在模拟中学习这种基于视觉的基于视觉的代理的选择，以及对真实机器人的最佳转移产生了什么影响。然后，我们利用这些策略收集的数据并通过离线RL改善它们。我们工作的视频和博客文章作为补充材料提供。
translated by 谷歌翻译
Neural networks leverage robust internal representations in order to generalise. Learning them is difficult, and often requires a large training set that covers the data distribution densely. We study a common setting where our task is not purely opaque. Indeed, very often we may have access to information about the underlying system (e.g. that observations must obey certain laws of physics) that any "tabula rasa" neural network would need to re-learn from scratch, penalising performance. We incorporate this information into a pre-trained reasoning module, and investigate its role in shaping the discovered representations in diverse self-supervised learning settings from pixels. Our approach paves the way for a new class of representation learning, grounded in algorithmic priors.
translated by 谷歌翻译
仿真最近已成为深度加强学习，以安全有效地从视觉和预防性投入获取一般和复杂的控制政策的关键。尽管它与环境互动直接关系，但通常认为触觉信息通常不会被认为。在这项工作中，我们展示了一套针对触觉机器人和加强学习量身定制的模拟环境。提供了一种简单且快速的模拟光学触觉传感器的方法，其中高分辨率接触几何形状表示为深度图像。近端策略优化（PPO）用于学习所有考虑任务的成功策略。数据驱动方法能够将实际触觉传感器的当前状态转换为对应的模拟深度图像。此策略在物理机器人上实时控制循环中实现，以演示零拍摄的SIM-TO-REAL策略转移，以触摸感的几个物理交互式任务。
translated by 谷歌翻译
We introduce a conceptually simple and scalable framework for continual learning domains where tasks are learned sequentially. Our method is constant in the number of parameters and is designed to preserve performance on previously encountered tasks while accelerating learning progress on subsequent problems. This is achieved by training a network with two components: A knowledge base, capable of solving previously encountered problems, which is connected to an active column that is employed to efficiently learn the current task. After learning a new task, the active column is distilled into the knowledge base, taking care to protect any previously acquired skills. This cycle of active learning (progression) followed by consolidation (compression) requires no architecture growth, no access to or storing of previous data or tasks, and no task-specific parameters. We demonstrate the progress & compress approach on sequential classification of handwritten alphabets as well as two reinforcement learning domains: Atari games and 3D maze navigation.
translated by 谷歌翻译
The ability to learn tasks in a sequential fashion is crucial to the development of artificial intelligence. Neural networks are not, in general, capable of this and it has been widely thought that catastrophic forgetting is an inevitable feature of connectionist models. We show that it is possible to overcome this limitation and train networks that can maintain expertise on tasks which they have not experienced for a long time. Our approach remembers old tasks by selectively slowing down learning on the weights important for those tasks. We demonstrate our approach is scalable and effective by solving a set of classification tasks based on the MNIST hand written digit dataset and by learning several Atari 2600 games sequentially.
translated by 谷歌翻译